Systems and methods for cell membrane identification and tracking, and technique automation using the same

ABSTRACT

A system including a processor, and memory having stored thereon instructions that, when executed by the processor, control the processor to receive image data of a sequence of images, and a current image of the sequence of images being after a previous image in the sequence of images, each of the current and previous images including a cell, filter the current image to remove noise, iteratively deconvolve the filtered current image to identify edges of the cell within the current image based on determined edges of the cell within the previous image, and segment the deconvolved current image to determine edges of the cell within the current image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/116,192 filed 29 Aug. 2018, which claims the benefit of U.S.Provisional Patent Application No. 62/551,570 filed 29 Aug. 2017, theentire contents and substance of which are hereby incorporated byreference as if fully set forth below.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under Grant No. 1409422awarded by the National Science Foundation. The government has certainrights in the disclosure.

THE NAMES OF THE PARTIES TO A JOINT RESEARCH AGREEMENT

Not Applicable

SEQUENCE LISTING

Not Applicable

STATEMENT REGARDING PRIOR DISCLOSURES BY THE INVENTOR OR A JOINTINVENTOR

Not Applicable

BACKGROUND OF THE DISCLOSURE 1. Field of the Disclosure

Embodiments of the present disclosure generally relate to systems andmethods for cell membrane identification and tracking and, inparticular, cell membrane identification and tracking through usingre-weighted total variation dynamic filtering.

2. Background

Live cell imaging allows the monitoring of complex biophysical phenomenaas they happen in real time, which is beneficial in studying biologicalfunctions, observing drug action, or monitoring disease progression. Forthese experiments, tissue from an organ such as brain, heart, or liveris sliced and imaged while it is still alive. Fluorescence microscopy isoften used for live cell imaging, but is not always practical because itrequires the use of dyes or genetic engineering techniques to introducefluorophores into the sample. Instead, it is often desirable to imageunlabeled, otherwise optically transparent samples. This is often doneusing a phase contrast-enhancing technique such as differentialinterference contrast (DIC) microscopy.

DIC microscopy is widely used for observing unstained biological samplesthat are otherwise optically transparent. Using machine vision toautomatically segment individual cells under DIC optics in real timewould be highly useful for microscopy automation of life scienceexperiments. However, precise cell segmentation is challenging, and therelated art algorithms are not directly applicable to segmentation underDIC in tissue. Further, identifying relevant features under DIC ischallenging. For instance, related art solutions fail to recognizecell-membrane boundaries with sufficient fidelity. Accordingly, certainlaboratory techniques have not been able to be efficiently automated.

For example, patch clamping is an important experimental technique oftaking high-fidelity electrophysiological measurements from singlecells; not only is it considered the “gold” standard for recordings ofelectrical potentials, it also allows for intercellular access (i.e.,extracting or inserting chemical and biological material from it). Onemethod of patch clamping requires a micropipette to be positionedadjacent to a cell's membrane, for a tight seal to be formed usingsuction between its tip and the cell's membrane. The sheer laboriousnessof such manual patch clamping makes automating this process highlyadvantageous.

Whole-cell patch clamp electrophysiology of neurons in vivo enables therecording of electrical events in cells with great precision andsupports a wide diversity of morphological and molecular analysisexperiments important for the understanding of single-cell and networkfunctions in the intact brain. In a typical whole-cell patch clampelectrophysiology implementation, a glass pipette electrode is used togain electrical and molecular access to the inside of a cell. Thispermits high-fidelity recording of electrical activity in neuronsembedded within intact tissue, such as in brain slices, or in vivo.Whole-cell patch clamp recordings of the electrical activity of neuronsin vivo, which utilize the glass micropipettes to establish electricaland molecular access to the insides of neurons embedded in intacttissue, exhibit signal quality and temporal fidelity sufficient toreport synaptic and ion-channel mediated subthreshold events ofimportance for understanding how neurons compute, and how theirphysiology can be modulated by brain disorders or pharmaceuticals.Whole-cell patch clamping of cells in intact tissue also allows forinfusion of chemicals and extraction of cell contents. Molecular accessto the cell enables infusion of dyes for morphological visualization, aswell as extraction of cell contents for transcriptomic single-cellanalysis, thus enabling integrative analysis of molecular, anatomical,and electrophysiological information about single neurons in intacttissue. Generally, the patch clamping process is extremely delicate, andthe micropipette must be placed just at the cell's boundary, and withintolerances of the robotic actuator (1-2 μm).

It is typical for in vitro brain slice electrophysiology to use DICmicroscopy since it is an intrinsic contrast microscopy that does notcause photo-bleaching and phototoxicity, in contrast to exogenouscontrast methods like fluorescence microscopy. Cell segmentation on DICmicroscopy is especially difficult due to the presence of DIC opticalartifacts. Prior work in the DIC cell segmentation literature were foundunsuitable in meeting the needs of this particular application. Forexample, traditional image processing methods for segmentation wereapplicable only on imagery with extremely low noise. The moresophisticated method of deconvolution was sought since it was able tohandle high-noise applications. Other promising and recent variants ofDIC deconvolution algorithms exploited structures of smoothness,sparsity, and dynamics for non-linear deconvolution, but they too werenot capable of handling the heavy organic interference experienced inthis data.

Related-art general purpose segmentation algorithms in the computervision literature typically assume statistical homogeneity within (oroutside) a segmentation region that is lost under contrast-enhancingoptical approaches such as DIC. Certain related-art cell segmentationand tracking methods are also not directly applicable to cellsegmentation under DIC in tissue. For example, some algorithms aredeveloped on the CTC dataset that includes only of microscopy imagery ofcultured cells (rather than tissue slices) that had minimal organictissue interference (FIG. 1). These cell-tracking methods (1) oftenassume simple noise statistics, (2) target gross cell location trackingfor mechanobiology tasks (e.g., studies on cell migration, morphology)rather than precise membrane localization, and (3) are designed to berun offline rather than in real time.

Recent work on patch clamping has had some success in using a roboticactuator to maneuver the pipette to form the seal (see, e.g., U.S. Pat.No. 9,668,804, US Patent Publication Nos. 2018/0028081, and US2017/0138926, the disclosures of which are incorporated herein byreference in their entireties). While the related art has demonstratedthe possibility of automating the patch clamp process by using amotorized robotic actuator to maneuver the probe to the target cell,such work has been limited by the related-art methods that do notsufficiently identify and track cell boundaries in real time. Forexample, several challenges that make cell membrane localization verydifficult (e.g., (1) heavy interference from the presence of organictissue around the target cell, (2) low signal-to-noise ratio (SNR) dueto scattering of light characteristic of thick tissue samples, and (3)cell motion induced by the glass probe) are not effectively overcome inthe related art.

What is needed, therefore, is a method and system for identifying andtracking cell membranes in high-noise samples that are capable ofprecisely localizing the cell membranes and performing the localizationin real time. It is to such a method and system that embodiments of thepresent disclosure are directed.

BRIEF SUMMARY OF THE DISCLOSURE

As described herein, there is a great need in the art to identifytechnologies for identifying and tracking cell membranes and use thisunderstanding to develop novel systems and methods for localizing cellmembranes with high precision in real time in samples with heavy organicinterference. Certain aspects of the present disclosure satisfy this andother needs.

According to some embodiments there is provided a system including aprocessor, and memory having stored thereon instructions that, whenexecuted by the processor, control the processor to receive image dataof a sequence of images, and a current image of the sequence of imagesbeing after a previous image in the sequence of images, each of thecurrent and previous images including a cell, filter the current imageto remove noise, iteratively deconvolve the filtered current image toidentify edges of the cell within the current image based on determinededges of the cell within the previous image, and segment the deconvolvedcurrent image to determine edges of the cell within the current image.

The instructions may further control the processor to identify, withinthe current image, a subset of the current image containing the cell,deconvolving being performed only the identified subset of the currentimage.

The instructions may control the processor to iteratively deconvolve thecurrent image by determining an estimated edge that minimizes a costfunction, adjusting weights of the cost function based on the estimatededge, and repeating the determining and adjusting.

The cost function may include a first time corresponding to a predictiveerror between the current image and an image predicted from theestimated edge, and a second term corresponding to a connectivitydetermination of the estimated edge within the current image.

The adjusted weights of the cost function may modify the connectivitydetermination of the estimated edge within the current image. Thedetermined edges of the cell within the previous image may impact theadjusted weights.

The instructions may control the processor to iteratively deconvolve thecurrent image by iteratively estimate an edge that minimizes a costfunction utilizing alternating direction method of multipliers (ADMM),adjusting weights of the cost function based on the estimated edge, andrepeating the determining and adjusting.

The instructions may further control the processor to outputinstructions to an articulating arm to position an instrument proximalto the determined edge of the current image. The instrument may includeone or more of an electrode, an injector, and a manipulation instrument.

The instructions may further control the processor to set the currentimage as the previous image, set a next image in the sequence of imagesas the current image, and repeat the filtering, iterativelydeconvolving, and segmenting.

The sequence of image comprises a live video, and the edges of the cellwithin the current image are determined in near real-time.

According to some embodiments there is provided a method includingreceiving image data of a sequence of images, and a current image of thesequence of images being after a previous image in the sequence ofimages, each of the current and previous images including a cell,filtering the current image to remove noise, iteratively deconvolvingthe filtered current image to identify edges of the cell within thecurrent image based on determined edges of the cell within the previousimage, and segmenting the deconvolved current image to determine edgesof the cell within the current image.

The method may further include identifying, within the current image, asubset of the current image containing the cell, deconvolving beingperformed only the identified subset of the current image.

The method may include iteratively deconvolving the current image bydetermining an estimated edge that minimizes a cost function, adjustingweights of the cost function based on the estimated edge, and repeatingthe determining and adjusting.

The cost function may include a first time corresponding to a predictiveerror between the current image and an image predicted from theestimated edge, and a second term corresponding to a connectivitydetermination of the estimated edge within the current image.

The adjusted weights of the cost function may modify the connectivitydetermination of the estimated edge within the current image, and thedetermined edges of the cell within the previous image may impact theadjusted weights.

The method may include deconvolving the current image by iterativelyestimate an edge that minimizes a cost function utilizing ADMM,adjusting weights of the cost function based on the estimated edge, andrepeating the determining and adjusting.

The method may further include moving an articulating arm to position aninstrument proximal to the determined edge of the current image.

The method may further include setting the current image as the previousimage, setting a next image in the sequence of images as the currentimage, and repeating the filtering, iteratively deconvolving, andsegmenting.

The sequence of image comprises a live video, and the edges of the cellwithin the current image are determined in near real-time.

According to some embodiments there is provided one or morenon-transitory computer-readable media comprising instructions that whenexecuted by one or more computing devices cause the one or morecomputing devices to receive from an imaging device image datarepresentative of a temporal sequence of images of at least a portion ofa cell, the temporal sequence of images including a current image and aprevious image imaged before the current image, filtering at least aportion of the noise from the current image, iteratively deconvolvingthe filtered current image to identify an edge of the cell within thefiltered current image based on a determined edge of the cell within theprevious image, and segmenting the deconvolved filtered current image todetermine the edge of the cell within the filtered current image.

These and other objects, features and advantages of the presentdisclosure will become more apparent upon reading the followingspecification in conjunction with the accompanying description, claimsand drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying Figures, which are incorporated in and constitute apart of this specification, illustrate several aspects described below.The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 depicts example images obtained with DIC microscopy (40×magnification). Left panel: cultured human embryonic kidney (HEK293)cell. Right panel: neuron in mouse brain tissue (400 μm thickness). Theimage on the right has high levels of imaging noise due to light scatterand interference due to organic material in surrounding tissue. Scalebar 10 am.

FIG. 2 depicts the example stages of the described imaging system. Thefull imaging system has three stages: patch tracking, deconvolution, andsegmentation. In patch tracking, a user-defined template is provided fortemplate matching and tracking spatial coordinates over time with aKalman filter. In deconvolution, the patch is deconvolved to infer theoptical path length (OPL) approximation. In segmentation, a globalthreshold is applied to yield a binary segmentation mask to determinethe cell membrane location.

FIG. 3 shows radially averaged power spectral density (RAPSD) plots inlog-scale from an example 64×64 cell patch. The Wiener filter utilizedan approximation to the signal (in blue) and the signal-with-noise (inred) power spectral density modeled as a linear polynomial least-squarefitting over the RAPSD. The signal approximation was computed byaveraging the power spectral density of simulated cells. Thesignal-with-noise approximation was extracted from an observed patch.The final filtered spectrum is shown in black.

FIG. 4 shows the contribution of pre-filtering seen asinterference-suppression. The first column of images pertains to theground truth simulation. The second and third columns pertain topre-filtering (PF)+re-weighted total variation dynamic filtering(RWTV-DF) (collectively, PF+RWTV-DF) to be performed without and withthe pre-filter respectively. While the pre-filter blurred high frequencyinterference content, it also blurred the cell's edges. The pre-filteredobservation has its edge integrity retained while suppressingsurrounding interference found at the bottom left corner and immediateright of the cell; this results in an overall segmentation that iscloser to the ground truth.

FIG. 5 shows a graphical model depicting the hierarchical Laplacianscale mixture model's Bayesian prior dependencies in the RWTV-DFalgorithm. Prior state estimates of the signal edges were used to setthe hyper-priors for the second level variables (i.e., variances of thestate estimates), thereby implementing a dynamical filter thatincorporates edge information into the next time step.

FIG. 6 shows the ground truth image that was the binary mask of asimulated image. Subsequent images reflected the iterations ofreweighting process during deconvolution (iterations 1 through 5respectively). Over the iterations, dominant edges were enhanced whileweaker ones recede, resulting in piecewise smooth solutions that areamenable to segmentation.

FIG. 7 shows a block diagram of the cell simulator. Gray blocks:generative models that learn from available data. White blocks:generative models that rely on user-parameters. The respective imagesare outputs from the various stages of the simulator, including (i)Binary image of synthetic cell shape (ii) Textured OPL image showinglight transmission through tissue (iii) Received image through DICoptics and (iv) Final image with noise and interference.

FIG. 8 shows quantile-quantile plots visualized similarities between thecharacteristics of simulated and actual cell-shapes. Distributions werecompared for the following shape-features: aspect ratio (top left), formfactor (top right), convexity (bottom left), and solidity (bottom right)as defined in the text. (c-1, c-2, d-1, d-2) Several outlier pairs fromthe convexity and solidity plots are identified and shown. The outlierpairs appeared to be caused by actual cell shape outlier statistics thatare underrepresented (due to lack of training examples) and thus notfully captured by the model.

FIGS. 9A-9B show the general qualitative similarity between randomlydrawn samples of (FIG. 9A) synthetically generated DIC cell images usingthe proposed simulator, and (FIG. 9B) actual DIC microscopy images ofrodent neurons. Similarities in imagery characteristics included: (1)DIC optical features as shown by the three-dimensional (3D) relief whichwere highlighted by the high/low intensity ‘shadows’ in the cell edges,(2) organic interference which appeared as Gaussian noise with afrequency profile, (3) cell shapes as demonstrated by the cell shapes'organic and natural contours, and (4) uneven lighting bias as shown bythe smooth but uneven background gradient.

FIG. 10 shows boundary error metrics. Each blue line l_(a) represents anapproach direction of a pipette towards the ground truth centroid.p_(a,B) represents the point where the algorithm thinks the membrane is.Individual boundary errors are computed as the distance ∥p_(a,B)−p_(a)∥₂and represented as black dotted lines. The average of all errorsconstitutes the average boundary error (ABE), the maximum constitutesthe maximum boundary error (MBE), and the variance of errors constitutesthe variance of boundary errors (VBE).

FIG. 11 shows average boundary errors ABE, MBE, and VBE for each of thealgorithms across 100 video trials of simulated data. ABE/MBE/VBE wasaggregated from 100 frames of 100 trials (i.e., 10,000 data points). ThePF+RWTV-DF algorithm was found to have a state-of-the-art performance,showing statistically significant improvements in ABE/MBE/VBE comparedto other algorithms (p≤0.001, denoted by the three stars based on pairedt-tests).

FIGS. 12A-12E show ABE, MBE, and VBE evolving with time are shown from(FIG. 12A) to (FIG. 12C) (respectively) for a single representativevideo (Trial #4) of a simulated DIC microscopy cell with 100 videoframes after deconvolution and segmentation using four differentalgorithms. The images in (FIG. 12D) are snapshots of the deconvolutionoutput at frame 100, and the images in (FIG. 12E) are their respectivesegmentations.

FIGS. 13A-13C show three example image patches of real cell data weredeconvolved using four algorithms and the resulting segmentations (toprow of FIGS. 13A, 13B and 13C) and deconvolutions (bottom row of FIGS.13A, 13B and 13C) are displayed. Segmentations were obtained fromdeconvolution by global thresholding. The proposed PF+RWTV-DF algorithmperforms consistently well even in severe interference (defined by thedegree of distortion around the edges and surface undulations within thecell).

FIGS. 14A-14C depict snapshots from real video data of a cell undergoingpatch clamping that demonstrated the efficacy of the proposed pipetteremoval method (via inpainting). The pipette mask is outlined in redwhile the cell segmentation is outlined in blue. Each snapshotillustrated varying degrees of overlap between the pipette and the cell:(FIG. 14A) the pipette is far from the cell, (FIG. 14B) the pipette isin close proximity to the cell, and (FIG. 14C) the pipette is “touching”the cell. The efficacy of the proposed inpainting method for pipetteremoval (versus no inpainting) is qualitatively demonstrated by itsability to cleanly segment the cell despite the pipette's presence.

FIGS. 15A-15B show the visual similarities that were observed betweenthe (FIG. 15A) synthetically generated cell-shapes, and the (FIG. 15B)cell-shapes extracted from DIC imagery of rodent brain slices.

FIGS. 16A-16D show (FIG. 16A) synthetically generated organic noise(synthetic) (FIG. 16B) An image patch of organic noise from a real DICimage (i.e. an image patch with no cell, only noise) (FIG. 16C)Comparisons of pixel intensity distributions (FIG. 16D) Comparisons ofthe RAPSD.

FIG. 17 shows four snapshots in time (indexed by k) from a synthesizedvideo (of 100 frames), generated from a single cell image. The cell'smotion induced by external forces (i.e., pipette motion, though notexplicitly present) is simulated by a slight contraction followed byexpansion over time, while performing a linear translation, from left toright of the frame. Synthetic interference (simulating organic material)shows up as a high-intensity blob around the bottom left corner of thecell, interfering with the cell's edges.

FIG. 18 illustrates deconvolution with inpainting for a single cell withdifferent iterations of ADMM between reweighing according to an exampleimplementation of the present disclosure.

FIG. 19 shows aggregated statistics of RWTV-ADMM across for differentiterations of ADMM between reweighing over 100 trials according to anexample implementation of the present disclosure.

FIG. 20 illustrates an environment in which one or more aspects of thepresent disclosure may be implemented.

FIG. 21 is a block diagram of an example computer system capable ofimplementing certain aspects of the present disclosure.

DETAILED DESCRIPTION OF THE DISCLOSURE

To facilitate an understanding of the principles and features of thevarious embodiments of the disclosure, various illustrative embodimentsare explained below. Although exemplary embodiments of the disclosureare explained in detail, it is to be understood that other embodimentsare contemplated. Accordingly, it is not intended that the disclosure islimited in its scope to the details of construction and arrangement ofcomponents set forth in the following description or examples. Thedisclosure is capable of other embodiments and of being practiced orcarried out in various ways. Also, in describing the exemplaryembodiments, specific terminology will be resorted to for the sake ofclarity.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferences unless the context clearly dictates otherwise. For example,reference to a component is intended also to include composition of aplurality of components. References to a composition containing “a”constituent is intended to include other constituents in addition to theone named. In other words, the terms “a,” “an,” and “the” do not denotea limitation of quantity, but rather denote the presence of “at leastone” of the referenced item.

As used herein, the term “and/or” may mean “and,” it may mean “or,” itmay mean “exclusive-or,” it may mean “one,” it may mean “some, but notall,” it may mean “neither,” and/or it may mean “both.” The term “or” isintended to mean an inclusive “or.”

Also, in describing the exemplary embodiments, terminology will beresorted to for the sake of clarity. It is intended that each termcontemplates its broadest meaning as understood by those skilled in theart and includes all technical equivalents which operate in a similarmanner to accomplish a similar purpose. It is to be understood thatembodiments of the disclosed technology may be practiced without thesespecific details. In other instances, well-known methods, structures,and techniques have not been shown in detail in order not to obscure anunderstanding of this description. References to “one embodiment,” “anembodiment,” “example embodiment,” “some embodiments,” “certainembodiments,” “various embodiments,” etc., indicate that theembodiment(s) of the disclosed technology so described may include aparticular feature, structure, or characteristic, but not everyembodiment necessarily includes the particular feature, structure, orcharacteristic. Further, repeated use of the phrase “in one embodiment”does not necessarily refer to the same embodiment, although it may.

Ranges may be expressed herein as from “about” or “approximately” or“substantially” one particular value and/or to “about” or“approximately” or “substantially” another particular value. When such arange is expressed, other exemplary embodiments include from the oneparticular value and/or to the other particular value. Further, the term“about” means within an acceptable error range for the particular valueas determined by one of ordinary skill in the art, which will depend inpart on how the value is measured or determined, i.e., the limitationsof the measurement system. For example, “about” can mean within anacceptable standard deviation, per the practice in the art.Alternatively, “about” can mean a range of up to ±20%, preferably up to±10%, more preferably up to ±5%, and more preferably still up to ±1% ofa given value. Alternatively, particularly with respect to biologicalsystems or processes, the term can mean within an order of magnitude,preferably within 2-fold, of a value. Where particular values aredescribed in the application and claims, unless otherwise stated, theterm “about” is implicit and in this context means within an acceptableerror range for the particular value.

Throughout this disclosure, various aspects of the disclosure can bepresented in a range format. It should be understood that thedescription in range format is merely for convenience and brevity andshould not be construed as an inflexible limitation on the scope of thedisclosure. Accordingly, the description of a range should be consideredto have specifically disclosed all the possible subranges as well asindividual numerical values within that range. For example, descriptionof a range such as from 1 to 6 should be considered to have specificallydisclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numberswithin that range, for example, 1, 2, 2.7, 3, 4, 5, 5.3, and 6. Thisapplies regardless of the breadth of the range.

Similarly, as used herein, “substantially free” of something, or“substantially pure”, and like characterizations, can include both being“at least substantially free” of something, or “at least substantiallypure”, and being “completely free” of something, or “completely pure”.

By “comprising” or “containing” or “including” is meant that at leastthe named compound, element, particle, or method step is present in thecomposition or article or method, but does not exclude the presence ofother compounds, materials, particles, method steps, even if the othersuch compounds, material, particles, method steps have the same functionas what is named.

Throughout this description, various components may be identified havingspecific values or parameters, however, these items are provided asexemplary embodiments. Indeed, the exemplary embodiments do not limitthe various aspects and concepts of the present disclosure as manycomparable parameters, sizes, ranges, and/or values may be implemented.The terms “first,” “second,” and the like, “primary,” “secondary,” andthe like, do not denote an order, quantity, or importance, but ratherare used to distinguish one element from another.

It is noted that terms like “specifically,” “preferably,” “typically,”“generally,” and “often” are not utilized herein to limit the scope ofthe claimed disclosure or to imply that certain features are critical,essential, or even important to the structure or function of the claimeddisclosure. Rather, these terms are merely intended to highlightalternative or additional features that may or may not be utilized in aparticular embodiment of the present disclosure. It is also noted thatterms like “substantially” and “about” are utilized herein to representthe inherent degree of uncertainty that may be attributed to anyquantitative comparison, value, measurement, or other representation.

The dimensions and values disclosed herein are not to be understood asbeing strictly limited to the exact numerical values recited. Instead,unless otherwise specified, each such dimension is intended to mean boththe recited value and a functionally equivalent range surrounding thatvalue. For example, a dimension disclosed as “50 mm” is intended to mean“about 50 mm.”

It is also to be understood that the mention of one or more method stepsdoes not preclude the presence of additional method steps or interveningmethod steps between those steps expressly identified. Similarly, it isalso to be understood that the mention of one or more components in acomposition does not preclude the presence of additional components thanthose expressly identified.

The materials described hereinafter as making up the various elements ofthe present disclosure are intended to be illustrative and notrestrictive. Many suitable materials that would perform the same or asimilar function as the materials described herein are intended to beembraced within the scope of the disclosure. Such other materials notdescribed herein can include, but are not limited to, materials that aredeveloped after the time of the development of the disclosure, forexample. Any dimensions listed in the various drawings are forillustrative purposes only and are not intended to be limiting. Otherdimensions and proportions are contemplated and intended to be includedwithin the scope of the disclosure.

DIC microscopy is widely used for observing unstained biological samplesthat are otherwise optically transparent. Combining this opticaltechnique with machine vision could enable the automation of many lifescience experiments; however, identifying relevant features under DIC isoften challenging. In particular, precise tracking of cell boundaries ina thick (>100 m) slice of tissue has not previously been accomplished.Herein is described a novel deconvolution algorithm that identifies andtracks these membrane locations.

Certain implementations provide cell segmentation and boundary trackingfor DIC imagery of tissue slices. One of ordinary skill will recognizethat certain aspects of the present disclosure may be utilized invarious tasks. However, for ease of description, certain portions of thepresent disclosure are related to patch-clamp recording in brain tissue.In this experimental paradigm, brain tissue was sliced into 100-400 μmthick sections, each containing thousands of neurons embedded in a densebiological milieu. The slice was imaged with a microscope and glassprobe was inserted into the tissue to sample the electrical activity ofindividual cells.

According to some embodiments, deconvolution of an image of a cell isperformed to find a cell boundary. Certain aspects of the presentdisclosure may involve, as non-limiting examples, patch tracking,deconvolution, and segmentation. Patch tracking may include performingpattern recognition on portion of an image to identify a general arealikely to contain cell boundaries (e.g., a cell). In someimplementations, patch tracking may not be performed, and deconvolutionmay be performed on an entirety of an image or one or more subsetsthereof. In some cases, the PF may remove noise from an organic image(e.g., an image of a brain slice). The PF may be tuned to statisticalcharacteristics of a specific imaging technique and tissue sample and/ortype. Deconvolution may utilize two-stages, PF and RWTV-DF. In somecases, the deconvolution may be performed on pre-filtered or unfiltereddata. As discussed below in greater detail, the deconvolution mayidentify cell edges based on how well predicted edges describe the imagedata (e.g., least means square) and how connected the various edges are(e.g., by minimizing a summation of spatial derivatives of the image).The deconvolution may be applied to a sequence of images (e.g., video)and the derivatives may be weighted based on detected edges in a priorimage in the sequence. The deconvolution may be iteratively performed,adjusting weights based on predicted edges, and then re-predicting theedges.

Examples

The present disclosure is also described and demonstrated by way of thefollowing examples. However, the use of these and other examplesanywhere in the specification is illustrative only and in no way limitsthe scope and meaning of the disclosure or of any exemplified term.Likewise, the disclosure is not limited to any particular preferredembodiments described here. Indeed, many modifications and variations ofthe disclosure may be apparent to those skilled in the art upon readingthis specification, and such variations can be made without departingfrom the disclosure in spirit or in scope. The disclosure is thereforeto be limited only by the terms of the appended claims along with thefull scope of equivalents to which those claims are entitled.

Cell Membrane Tracking In Living Brain Tissue Using DIC Microscopy

The proposed algorithm extends the framework provided by others and wasformulated as a regularized least-squares optimization that incorporatesa filtering mechanism to handle organic tissue interference and a robustedge-sparsity regularizer that integrates dynamic edge-trackingcapabilities. The described algorithm performed a deconvolution of thecomplex effects of DIC imaging integrated into a segmentation processand was not a direct segmentation of raw DIC images. Toward this end,the focus of this algorithm was specifically on cell boundary trackinginstead of more typical deconvolution metrics such as least-square imagereconstruction. Also described herein is a MATLAB toolbox for accuratelysimulating DIC microscopy images of in vitro brain slices. Building onexisting DIC optics modelling, the simulation framework additionallycontributed an accurate representation of interference from organictissue, neuronal cell-shapes, and tissue motion due to the action of thepipette. This simulator allows users to better understand the imagestatistics to improve algorithms, as well as quantitatively test cellsegmentation and tracking algorithms in scenarios where ground truthdata is fully known. The simulation toolbox is freely available underGNU GPL at siplab.gatech.edu.

DIC Microscopy

DIC microscopy can enhance the contrast of an image by exploiting thefact that differences in the tissue often have different opticaltransmission properties that can be measured through the principle ofinterferometry. Specifically, the signal that is reconstructed by thealgorithm is known as the OPL signal image, which is proportional to theunderlying phase-shift. The OPL is defined as the product of refractiveindex and thickness distributions of the object relative to thesurrounding medium. This OPL signal was denoted as x_(k)∈

^(N) (a vectorized form of the √N×√N OPL image's intensity values) andindexed in time by subscript k. DIC microscopy was used to amplifydifferences between the cell's refractive index and its environment toenhance visibility of the cell, thus highlighting edge differentials andgiving the appearance of 3D relief. This effect can be idealized as aconvolution between the optics' point spread function and the OPL, anddenoted as Dx_(k), where D∈

^(N×N) is a matrix that captures the two-dimensional (2D) convolutionagainst a kernel d∈

^(K×K).

While more sophisticated DIC imaging models exist, for means of example,a kernel d corresponding to an idealized DIC model which is a steerablefirst derivative of Gaussian kernel may be used:

$\begin{matrix}{{d( {x,y} )} \propto {{{- x}\;{{\cos(\theta)} \cdot e^{- \frac{x^{2} + y^{2}}{\sigma_{d}^{2}}}}} - {y\;{{\sin(\theta)} \cdot e^{- \frac{x^{2} + y^{2}}{\sigma_{d}^{2}}}}}}} & (1)\end{matrix}$

where σ_(d) refers to the Gaussian spread and θ_(d) refers to asteerable shear angle. This model assumed an idealized effective pointspread function (EPSF), where the condenser lens is infinite-sized andthe objective is infinitely small (with respect to the wavelength). Thismodel also ignored phase wrapping phenomenons by assuming that thespecimen is thin enough or the OPL varies slowly enough such that itbehaves in the linear region of the phase difference function. Inpractice, d(x, y) is discretized as d with a proportionality constant of1, and with x and y each taking the discrete domain

$\begin{matrix}\{ {{- \frac{K - 1}{2}},{- \frac{K - 3}{2}},\ldots\mspace{11mu},\frac{K - 3}{2},\frac{K - 1}{2}} \} & (2)\end{matrix}$

DIC cell segmentation algorithms generally fall into three categories:direct, machine-learned, or deconvolution (or phasereconstruction)algorithms. Direct algorithms apply standard image processing operationssuch as low-pass filtering, thresholding, and morphological shapeoperations, but often are not robust and commonly work optimally on verylow-noise/interference imagery. Machine-learned algorithms performstatistical inference learned from a large number of cell-specifictraining images (e.g., deep convolutional networks or Bayesianclassifiers). Such algorithms have been shown to perform coarsesegmentation surprisingly well in challenging scenarios (e.g., cellswith complicated internal structures with low-noise/interference) forapplications like cell-lineage tracking or cell-counting, yet theyappear to lack precision for accurate cell-boundary localization.Certain related art deconvolution algorithms may be effective atretrieving the OPL for noisy microscopy images with little interference.However, organic tissue interference surrounding the cell negativelyaffected the reconstruction (and subsequently segmentation), especiallyaround the edges of the cell.

Meanwhile, a reweighted ι1 aframework was found to produce robustreconstructions because each signal element's statistics wereindividually parameterized as opposed to having them globallyparameterized by a single term (in the non-weighted 1 setup). Thismethod, though very effective for the tracking of sparsely distributedsignals, may not be directly applied since the signal space (i.e., theDIC imagery) distribution is inadequately sparse. Likewise, thereweighted TV minimization as a regularization method (in compressivesensing), for recovering the Shepp-Logan phantom from sparse Radonprojections, demonstrating potential for application in bioimagery, isnot capable of exploiting dynamical information in temporal data.

One of ordinary skill will understand that this is merely an example,and aspects of the disclosure may be applied to various imaging models.

Cell Simulator

Accurate cell simulators are a valuable tool for two reasons: they allowobjective testing with known ground truth and provide insights intogenerative models that facilitate algorithm development. Currently, thevast majority of existing cell simulator packages focus specifically onfluorescence microscopy rather than DIC microscopy. While somesimulators excelled at providing a large variety of tools for simulatingvarious experimental scenarios and setups, most lacked simulation ofsynthetic cellular noise similar to that found in DIC microscopy imagesof brain slices (due to the presence of cellular tissue). Mostsimulators tend to target very specific types of cells and there havebeen initial efforts to organize and share cellular information (e.g.,spatial, shape distributions) into standardized formats acrosssimulators. Despite this, no existing simulator currently generatessynthetic DIC imaging of neurons such as those used in patch clampexperiments for brain slices. To facilitate algorithm design andevaluation on the important problem of automated patch clamping, aMATLAB toolbox has been built and released for accurately simulating DICmicroscopy images of in vitro brain slices.

Formulation of the Deconvolution Algorithm

Certain aspects of the described systems and methods are designed toprovide automated visual tracking of the membrane of a user-selectedtarget cell (e.g., to guide a robotic patch clamping system). Accordingto some embodiments, three general stages visualized by FIG. 2 are used.The first stage includes computer vision tracking techniques to identifythe general patch of interest containing the target cell on the currentframe (e.g., a current image of a video containing the cell). The secondstage implements dynamic deconvolution to recover a time varying OPLimage that can be used for segmentation. In some cases, good performanceis retained when a foreign object (e.g., a recording pipette) overlapswith the cell by removing the foreign object from the image andperforming inpainting. The third stage includes a segmentation on theoutput of the deconvolution, which was performed with simplethresholding. It is suggested that a simple segmentation strategy issufficient after a robust deconvolution process.

Through the first stage, the deconvolution and segmentation algorithmsmay be run on a limited image patch size (e.g., 64×64, as a trade-offbetween speed and resolution) that was found by tracking gross motion inthe image. For example, template matching (via normalizedcross-correlation) may be performed on each frame using a user-selectedpatch (e.g., obtained via a mouse-click to associate coordinates of theimage patch center with the targeted cell's center). Patch-trackingrobustness may be improved by running a standard Kalman filter on thepositional coordinates of the found patch (i.e., tracking atwo-dimensional state vector of the horizontal and vertical location ofthe patch center). To increase efficacy with the correlation approach, apre-processing stage may be used, for example, to eliminate lightingbias (i.e., uneven background subtraction) on each video frame. In somecases, quadratic least squares estimation (with a polynomial order of 2)may be used, but this is merely an example. In some cases, biaselimination may be applied to an entire video frame (rather than to eachpatch) for computational efficiency. Also, if the bias conditions werestatic throughout a video or multiple frames, the bias or other PF maybe computed once, cached, then applied across multiple frames. Whiletracking the detailed cell membrane locations is challenging in therelated art, tracking the general location of the patch containing thetarget cell can be done with very high accuracy.

Formulation of the PF+RWTV-DF Algorithm

Certain aspects of the present disclosure relate to a PF+RWTV-DFalgorithm constructed of two distinct components: a PF operation, andthe RWTV-DF deconvolution algorithm. The two parts played distinct yetimportant roles: PF performed interference suppression while the RWTV-DFdeconvolution achieved dynamic edge-sparse reconstructions. However,this is merely an example, and, in some cases, the RWTV-DF may beutilized without PF.

(1) PE: The presence of non-white Gaussian noise due to organic tissueinterference caused challenges in recognizing membrane boundaries. Insome instances, PDF may be used to estimate the interference-freeobservation, for example, by “whitening” the spectra associated with theobservation and amplifying the spectra associated with the signal.Specifically, in some cases, the PF filter's design was derived usingthe Wiener filter and given in the spatial-frequency domain by:

$\begin{matrix}{{F_{k}}^{2} = {\frac{{{\mathcal{F}{\{ d \} \cdot {\hat{X}}_{k}}}}^{2}}{{{\mathcal{F}{\{ d \} \cdot {\hat{X}}_{k}}}}^{2} + {{\hat{N}}_{k}}^{2}} = \frac{{{\mathcal{F}{\{ d \} \cdot {\hat{X}}_{k}}}}^{2}}{{{\hat{Y}}_{k}}^{2}}}} & (3)\end{matrix}$

where F_(k) is the spatial Fourier spectrum of the pre-filter,

{d} is the spatial Fourier transform of the DIC imaging function fromEquation 1, X_(k) is the OPL signal's spatial Fourier spectrum estimate,N_(k) is the spatial Fourier spectrum estimate of n_(organic) ^(k)combined with n_(sensor) ^(k) and Ŷ^(k) is the signal-plus-noise'sspatial Fourier spectrum estimate. All spatial Fourier spectra wereestimated using least-squares polynomial fits of the RAPSD, defined as aradial averaging of the DC-centered spatial-frequency power spectrumdensity. Intuitively, this estimated an image's underlying spectrum viaa form of direction-unbiased smoothing. Specifically, the signal-onlycomponent X_(k) was estimated from averaged RAPSDs from OPLs ofsimulated cells generated from the realistic simulation frameworkdescribed herein, and estimated the combined signal-plus-noise spectraŶ_(k) from the RAPSDs of observed images y_(k). FIG. 3 illustrates theestimation process and effect of the pre-filter in the frequency domain.The positive contribution of the prefilter was highlighted in FIG. 4 inthe segmentation algorithm described below.

(2) RWTV-DF

Given the edge-sparse nature of the data and the particular need foraccuracy in the cell membrane locations during segmentation (in contrastto the more typical MSE minimization of the deconvolved image), the TVnorm may be used as the core regularization approach for this algorithm.Specifically, in some cases, the calculated isotropic TV may be takenas:

$\begin{matrix}{{{x_{k}}_{TV} = {{\sum\limits_{i = 1}^{N}{{T_{i}x_{k}}}_{2}} = {\sum\limits_{i = 1}^{N}{t_{k}\lbrack i\rbrack}}}},} & (4)\end{matrix}$

where T_(i)∈

^(2×N) is an operator that extracts the horizontal and verticalforward-differences of the i-th pixel of x_(k) into an

² vector, whose

² norm is denoted as an individual edge-pixel t_(k)[i]. This basicregularizer was improved via an iterative estimation of weights on theindividual edges through a Majorize-Minimization algorithm. However,this regularizer does not incorporate dynamical information for thetracking of moving edges.

To address this issue, a hierarchical Bayesian probabilistic model wheredynamics are propagated via second-order statistics from one-time stepto the next may be used designed such that the described method operatessolely in the edge-pixel space. As will be understood in light of thepresent disclosure, the sparse edge locations at the previous frameprovided strong evidence that there could be an edge nearby in thecurrent frame. Edge locations may be modeled with a sparsity-inducingprobability distribution with a parameter controlling the spread (i.e.,variance) of the distribution. When previous data gave evidence for anedge in a given location, the parameter to increase the variance of theprior data in this location, thereby likely making it easier for theinference to identify the presence of the edge from limitedobservations. In contrast, when previous data indicated that an edge ina location was unlikely, this variance may be decreased therebyrequiring more evidence from the observations to infer the presence ofan edge.

As a non-limiting example, at the lowest level of the hierarchy, thepre-filtered observations may be modeled conditioned on the OPL signalx_(k) with a white Gaussian distribution:

$\begin{matrix}{{p( {{F_{k}y_{k}}❘x_{k}} )} \propto e^{{- \frac{1}{2\;\sigma^{2}}}{{{F_{k}y_{k}} - D_{x_{k}}}}_{2}^{2}}} & (5)\end{matrix}$

where F_(k) is the matrix operator describing 2D convolution (in thefrequency domain) against the pre-filter F_(k) described earlier byEquation 4. At the next level, the individual edge-pixels t_(k)[i] maybe assumed sparse and therefore best modeled with a distribution withhigh kurtosis but with unknown variance. One of ordinary skill willrecognize that aspects of the present disclosure may be utilized fordifferent optic models (i.e., optic models besides DIC) by modifyingF_(k) to fit the different optical models. Conditioning on a weightingγ_(k)[i] that controlled the individual variances, the t_(k)[i] may bemodeled as random variables arising from independent Laplaciandistributions:

$\begin{matrix}{{p( {{t_{k}\lbrack i\rbrack}❘{\gamma_{k}\lbrack i\rbrack}} )} = {\gamma_{0}\;\frac{\gamma_{k}\lbrack i\rbrack}{2}e^{{- \gamma_{0}}{{\gamma_{k}{\lbrack i\rbrack}} \cdot {{t_{k}{\lbrack i\rbrack}}}}}}} & (6)\end{matrix}$

where γ₀ is a positive constant. At the top-most level, the weightsγ_(k)[i] are themselves also treated as random variables with a Gammahyperprior:

$\begin{matrix}{{p( {{\gamma_{k}\lbrack i\rbrack}❘{\theta_{k}\lbrack i\rbrack}} )} = {\frac{\gamma_{k}^{\alpha - 1}\lbrack i\rbrack}{{\theta_{k}^{a}\lbrack i\rbrack}{\Gamma(\alpha)}}e^{{- {\gamma_{k}{\lbrack i\rbrack}}}/{\theta_{k}{\lbrack i\rbrack}}}}} & (7)\end{matrix}$

where a is a positive constant,

(·) is the Gamma function, and θ_(k)[i] is the scale variable thatcontrols the Gamma distribution's mean and variance over γ_(k)[i]. Thistop-level variable may be used to insert a dynamics model into theinference that controlled the variance of the prior being used to inferedge locations based on previous observations. FIG. 5 illustrates themultiple prior dependencies of the hierarchical Laplacian scale mixturemodel described in Equations 5, 6, and 7 using a graphical model.

To build dynamics into the model, it was observed that the Gammadistribution's scale variable θ_(k)[i] controlled its expected value(i.e., E[γ_(k)][i]=αθ_(k)[i]). A dynamic filtering approach was thendesigned to the problem of interest by using a dynamics model on theedge-pixels that set the individual variances θ_(k)[i]'s according topredictions from the dynamics model g_(k)(·):

$\begin{matrix}{{\theta_{k}\lbrack i\rbrack} = \frac{\xi}{{{{g_{k}( t_{k - 1} )}\lbrack i\rbrack}} + \eta}} & (8)\end{matrix}$

where ξ is a positive constant and η is a small constant that preventsdivision by zero. To illustrate the operation of this model, a strongedge-pixel in a previous frame (i.e., a large value of t_(k−1)[i]) set asmall value of θ_(k)[i] (and respectively a small expected value ofγ_(k) [i]), in turn making the Laplacian's variance large (i.e.,Var[t_(k)[i]]=2/(γ₀γ_(k)[i])²). A large Laplacian variance implied ahigher likelihood that the edge-pixel in the current frame was active(in contrast to the reverse situation where a weak edge pixel wouldresult in a small Laplacian variance at the next time step). Therefore,this approach propagated second-order statistics (similar to classicKalman filtering for Gaussian models) through the hyperpriors γ_(k)using dynamic information via the evolution function g_(k)(·) at eachtime-step. One option for g_(k)(·) was a convolution against a Gaussiankernel (with its a proportional to expected motion variation), whichexpresses a confidence neighborhood of edge locations based on previousedge locations. As with any tracking algorithm, the tracking qualitydepended on the accuracy of the dynamics model and including bettermodels (e.g., more accurate motion speeds, motion direction informationbased on pipette movement, etc.) would improve the performance of anyapproach. With a fixed dynamics function, for each frame the algorithmtook a re-weighting approach where multiple iterations are used toadaptively refine the estimates at each model stage (illustrated for asimulated patch in FIG. 6).

The final optimization followed from taking the MAP estimate forEquations 5, 6, and 7 and applying the EM approach to iteratively updatethe weights. In some cases, the maximization step is given as a convexformulation

$\begin{matrix}\begin{matrix}{{\hat{x}}_{k}^{(t)} = {\arg\;{\min\limits_{x \geq 0}{- {\log\lbrack {p( {x❘\gamma_{k}^{(t)}} )} \rbrack}}}}} \\{= {{\arg\;{\min\limits_{x \geq 0}{\frac{1}{2}{{{F_{k}y_{k}} - {Dx}}}_{2}^{2}}}} + {\gamma_{0}{\sum\limits_{i = 1}^{N}{{\gamma_{k}^{(t)}\lbrack i\rbrack} \cdot {{{T_{i}x}}_{2}.}}}}}}\end{matrix} & (9)\end{matrix}$

In other words, {circumflex over (x)}_(k) ^((t)) is the value of x for agiven iteration (t) of a given image k (e.g., frame) that minimizes thecost function: ½∥F_(k)y_(k)−Dx∥₂ ²+γ₀ ^((t))Σ_(i=1) ^(N)γ_(k)^((t))[i]·∥T_(i)x∥₂. In light of the present disclosure, of ordinaryskill will recognize that the first portion of Equation 9(½∥F_(k)y_(k)−Dx∥₂ ²) represents the least mean square error of theprediction x^((t)) from the true image, while the second portion ofEquation 9 (γ₀ ^(t)Σ_(i=1) ^(N)γ_(k) ^((t))[i]·∥T_(i)x∥₂) represents thetotal derivative value for the image for a given prediction x^((t)). Thesecond portion describes how “connected” the predicted edges are in astatistical sense. The cost function balances the predicted error andthe “connectedness” of the prediction.

DIC deconvolution can be particularly sensitive to uneven (i.e.,non-matching) boundaries. Hence, the implementation of the matrixoperator D, though application dependent, should be treated carefully.In some embodiments, a discrete convolutional matrix implementation thathandled non-matching boundary could cause memory issues when N is largesince D scales quadratically in size (though it could be mitigated usingsparse matrices). Alternatively, in some embodiments, an FFT/IFFTsurrogate implementation implies circular boundary conditions that needto be explicitly taken care of (e.g., via zero-padding). The expectationstep may be using the conjugacy of the Gamma and Laplace distributions,admitting the closed-form solution

$\begin{matrix}{{\gamma_{k}^{({t + 1})}\lbrack i\rbrack} = {{{\mathbb{E}}_{p{({\gamma ❘{{\hat{t}}_{k}^{(t)}{\lbrack i\rbrack}}})}}\lbrack\gamma\rbrack} = \frac{\kappa + 1}{{\kappa \cdot {{{\hat{t}}_{k}^{(t)}\lbrack i\rbrack}}} + {{{g_{k}( {\hat{t}}_{k - 1} )}\lbrack i\rbrack}} + \eta}}} & (10)\end{matrix}$

where the Expectation Maximization (EM) iteration number is denoted bysuperscript t, t_(k) ^((t))[i]=∥T_(i)x_(k) ^((t))∥₂ is the estimate ofthe edge at iterate t, t_(k−1)[i]=∥T_(i)x_(k−1)∥₂ is the estimate of theedge at the previous time-step, γ₀ is the positive constant thatcontrolled the weight of edge-sparsity against reconstruction fidelity,and x is the positive constant that controls the weight of the currentobservation against the dynamics prior. The Gaussian convolutionimplemented by g_(k)(·) effectively “smeared” the previous time-step'sestimation of edges to form the current prior, accounting foruncertainty due to cell movement over time. In other words, g_(k)(·)ties the edge prediction of a current image to the edge prediction ofthe previous image in the sequence. The EM algorithm was initialized bysetting all weights to 1 (i.e., γ_(k) ⁽⁰⁾=1) and solving Equation 9,which is simply (non-weighted) TV-regularized least squares. Thereweighting iterations were terminated upon reaching some convergencecriteria (e.g., ∥x^((t))−x^((t−1))∥₂/∥x^((t))∥₂<ε, or the cost functionvalue for {circumflex over (x)}_(k) ^((t)) being less than a certainthreshold (which may work as a stand-in for fit) or after a fixed numberof reweights.

Object Removal via Inpainting

When an object (e.g., a pipette) draws near to a cell for interaction(e.g., patching), the object in the image patch can cause significanterrors in the deconvolution process. Accordingly, in some cases, thePF+RWTV-DF algorithm may be modified to accommodate objects within theimage. For example, an inpainting approach may mask away the object'spixels and inferred the missing pixels according to the same inverseprocess used for deconvolution. To begin, the object location may betracked in the image with a similar template matching process asdescribed above (which could be improved with positional informationfrom an actuator if available). Simultaneously, an associated objectmask (obtained either via automatic segmentation or manually drawn) wasaligned and overlaid on top the cell image patch using the updatedpipette locations. This object mask overlay may take the form of amasking matrix M∈{0, 1}^(M×N) whose rows are a subset of the rows of anidentity matrix, and whose subset is defined by pixel indices outsidethe pipette region. The original filtered observation F_(k)y∈

^(N) is reduced to M F_(k)y∈

^(M) (with M≤N) while the convolution operator becomes MD. Theinpainting version of Equation 9 is

$\begin{matrix}{{\hat{x}}_{k}^{(t)} = {{\arg\;{\min\limits_{x \geq 0}{\frac{1}{2}{{{m\; F_{k}y_{k}} - {MDx}}}_{2}^{2}}}} + {\gamma_{0}{\sum\limits_{i = 1}^{N}{{\gamma_{k}^{(t)}\lbrack i\rbrack}{{{T_{i}x}}_{2}.}}}}}} & (11)\end{matrix}$

Pixels are removed in the object region from the data used in thedeconvolution, but the algorithm may infer pixel values consistent withthe statistical model used for regularization.

Reweighting Modification

In some implementations, solving the deconvolution problem (i.e.,Equations 9 and 10 described above) at the current video frame inreal-time may be computationally expensive. In implementing thedescribed algorithm, the main computational bottleneck lies in solvingthe expensive optimization of Equation 9. In some implementations, ADMMmay be used to solve Equation 9. As will be understood by one ofordinary skill, ADMM framework is itself also iterative in nature, wherethe numerical precision of the method is proportional to the number ofiterations the solver carries out. For very high precision, the numberof iterations would easily run into the thousands. Thus, solvingEquation 9 with high precision (e.g., before reweighting in Equation 10)may take significant time, and request significant and computing power.However, in some implementations, RWTV-DF is merged with ADMM iterationsunder a single solver “umbrella”. By drastically reducing the time spenton producing high precision solutions in each iteration of Equation 9,and performing more rounds of reweighting, high precision may be had fora final edge estimation for a given frame without requiring highprecision estimations in each iteration.

FIG. 18 illustrates example edge determinations for a given frame fordifferent iterations of ADMM between reweighing T. Meanwhile, FIG. 19illustrates aggregated statistics of RWTV-ADMM for different iterationsof ADMM between reweighing T according to an example implementation.Thus, as can be seen, certain implementations may increase edgedetermination speed by utilizing lower fidelity while selectiveincreasing iteration times.

Results

Experimental Conditions

The performance of an implementation of the inventive algorithm wasevaluated by using a cell simulator that generates synthetic DIC imageryof rodent neurons in brain slices. Herein is shown the simulationframework and its realism is demonstrated by comparisons against data.

Most cell simulators can be described as having three distinct stages:cell-shape generation, optical imaging, and noise generation. Thesimulation framework implemented for this method used these same threestages (shown in FIG. 7), adapting general approaches used in theliterature for each stage so that the simulated data reflects thestatistics of the DIC microscopy images for in vitro brain sliceelectrophysiology. In the first stage, a synthetic cell-shape wasgenerated and embedded into the pixel-space as an ideal OPL image (i.e.,the ground truth). In the second stage, the OPL image was transformedusing a convolution against an idealized point spread function thatapproximately described the DIC microscope's optics. Lastly,interference from static components (e.g., organic material) and dynamicnoise components (e.g., the image acquisition system) are generated andincorporated into the image during the third stage. Herein, existinggeneral approaches were used to build and validate the model componentsin the simulator using real DIC imagery of in vitro brain slices fromadult (P50-P180) mice. This simulator is unique in that it is tailoredto simulate this type of cells. Full implementation details for thissimulator are discussed below.

To evaluate the realism of synthetically generated cell-shapes, severalof the shape features were compared to actual hand-drawn cell shapesfrom rodent brain slice imagery using four dimensionless shape features:

(1) Aspect Ratio was defined as the ratio of minor axis lengths to majoraxis lengths. The major/minor axis was determined from the best fitellipsoid of the binary image.

(2) Form factor (sometimes known as circularity) was defined as thedegree to which the particle is similar to a circle (taking intoconsideration the smoothness of the perimeter, defined as

$\frac{4\pi A}{P^{2}}$where A is area and P is the perimeter.

(3) Convexity was a measurement of the particle edge roughness, definedas

$\frac{P_{c\nu x}}{P}$where P_(cx) is the convex hull perimeter and P is the actual perimeter.

(4) Solidity was the measurement of the overall concavity of a particle,defined as

$\frac{A}{A_{c\nu x}}$where A_(cvx) is the convex hull area and A is the image area.

Using 50 shape-coordinates per cell, 115 synthetic cells were generated.The quartile-quartile plots in FIG. 8 showed that all four features aresimilarly distributed, demonstrating that the simulation rendersrealistic cell shapes for rodent in vitro brain slices.

Several simulation images were generated in FIG. 9 for visual comparisonagainst actual cell images. The following user-defined parameters (asdefined in Table I) were used: image size was 60×60, cell-rotation(θ_(rot)) was randomized, cell-scaling defined as factor of image widthwas γ_(scale)=0.8, DIC EPSF parameters were {θ_(d)=2250, σ_(d)=0.5},dynamic noise parameters were {Ag=0.98, σ_(g)=0.018, A_(p)=20.6,λ_(p)=10⁻¹⁰}, and SNR χ=−1 dB. Cell shapes were randomly generated basedon learned real cell-shapes, a lighting bias was taken randomly fromactual image patches, and organic noise was generated using a RAPSDcurve learned from real DIC microscopy images. In general, the syntheticand actual cells were qualitatively similar in cell shapes and noisetextures.

TABLE I User-Defined Simulator Properties Parameter Description M Imagesize (i.e., M × M pixel-image) θ_(rot) Rotation angle of cell-shapeγ_(scale) Scaling factor of cell-shape within image p Persistence of OPLsurface texture {σ_(d), θ_(d)} DIC imaging function parameters (Eq. (1)){A_(g), σ_(g)} Dynamic noise Gaussian parameters (Eq. (15)) {A_(p),λ_(p)} Dynamic noise Poisson parameters (Eq. (15)) χ SNR (in dB) offinal image (Eq. 16)

Segmentation of Synthetic Data

The implementation of the PF+RWTV-DF algorithm was tested on syntheticcell data and compared its performance against the state-of-the-artdeconvolution algorithms such as least-squares regularized by ι1 and TV(L1+TV), least-square regularized by ι1 and Laplacian Tikhonov (L1+Tik),and least-square by regularized re-weighted ι1, weighted LaplacianTikhonov, and weighted dynamic filtering (RWL1+WTik+WDF). Thedeconvolution from each algorithm is then segmented using a globalthreshold to produce a binary image mask for algorithmic evaluation.

For evaluation, three boundary metrics were employed that capturederrors relevant to the process of patch clamping. The metrics ABE, MBE,and VBE measured the (average, maximum, and variance of) distance errorsbetween the approximated cell boundary and the actual cell boundaryaround the entire membrane of the cell at each video frame.Specifically, these metrics captured the statistics of the distanceerrors between the actual and estimated cell boundaries as the pipettetip approaches the cell membrane in a straight path (directed towardsthe true centroid).

To compute these metrics, first potential pipette paths were discretizedtowards the true centroid using a set of lines {l_(a)}, defined as linesthat start at the true centroid and that infinitely extend through eachpoint p_(a) in the set of ground truth contour pixels {A}. Next,traveling along each l_(a) toward the centroid we found the intersectingpoint in the set of estimated contour pixels {B}, defined as p_(a,B)=argmin_(p) _(b) _(∈B)∥p_(b)−p_(l)∥₂ s.t. p_(l)∈l_(a), where are points onthe line l_(a). In other words, found were the points that the algorithmthat would identify as the membrane location during every hypotheticalapproach direction of the pipette. The ∥p_(b)−p_(l)∥₂ in the objectiveconsiders only orthogonal projections of p_(b) unto the line l_(a)(since it is a minimization), and it searches over all of B to find thep_(b) whose orthogonal projection distance is minimal. Note that thatdue to the contour discretization, pa,B does not lie exactly along l_(a)but rather close to it on some p_(b)∈B. With each discretized boundaryerror expressed as ε(a, B)=∥p_(a,B)−p_(a)∥₂, the metrics are thendefined as:

$\begin{matrix}{{{{ABE}( {A,B} )} = {\frac{1}{{A}_{0}}{\sum\limits_{a \in A}{ɛ( {a,B} )}}}}{{{MBE}( {A,B} )} = {\max\limits_{a \in A}{ɛ( {a,B} )}}}{{{VBE}( {A,B} )} = {\frac{1}{{A}_{0} - 1}{\sum\limits_{a \in A}( {{ɛ( {a,B} )} - {{ABE}( {A,B} )}} )^{2}}}}} & (12)\end{matrix}$

FIG. 10 illustrates the terms involved in these error metriccomputations.

100 videos of synthetic cell patches (each video contains 64×64pixels×100 frames) were generated using the proposed cell simulator withDIC EPSF parameters σ_(d)=0.5 and θ_(d)=2250 (assumed to be known eitherby visual inspection (to accuracy of ±15°) or more accurately by using acalibration bead). By physically relating real cells to simulated cells,the relationship between physical length and pixels was determined to beapproximately 3.75 pixels per 1.0 μm.

For the algorithms L1+TV, L1+Tik, and RWL1+WTik+WDF, the parameters forsparsity (β) and smoothness (γ) were tuned via exhaustive parametersweep on the first frame of the video (to find the closestreconstruction/deconvolution by the ι2 least squares sense).Specifically, the sweep was conducted over 10 logarithmically spacedvalues in the given ranges: L1+TV searched over [10⁻³, 10⁻⁹] for β and[10⁻¹, 10⁻⁵] for γ, L1+Tik searched over [10⁻³, 10⁻⁹] for β and [10⁻¹,10⁻⁷] for γ, and RWL1+WTik+WDF searched over [10⁻³, 10⁻⁹] for β and[10⁻¹, 10⁻⁵] for γ. L1+Tik (the non-reweighted version) was used in allexperiments because it was experimentally found to be better performingcompared to the reweighted version; reweighting over-induced sparsity ofpixels, which is not advantageous to segmentation in this particularapplication. For RWL1+WTik+WDF, the dynamics parameter was set to beδ=1.0×10⁻³, with a maximum of 80 reweighting steps (where convergence isfulfilled before each reweighting). For PF+RWTV-DF (Equations 9, 10),γ₀=3.0×10⁻⁴ and κ=5 was fixed, with four reweighting steps (whereconvergence is fulfilled before each reweighting). The programsspecified by L1+TV, RWL1+WTik+WDF, and PF+RWTV-DF were solved using CVX,while L1+Tik was solved using TFOCS.

FIG. 11 reflects statistics of the respective algorithms from each ofthe 100 frames from the 100 video trials. The height of each bar refersto the average of ABE/MBE/VBE while the error bars refer to ±1 standarderror of the ABE/MBE/VBE over the 100 video trials. PF+RWTV-DF was thebest performing algorithm in the three metrics, and there was asignificant difference (p≤0.001) between PF+RWTV-DF and the otheralgorithms for all metrics. Notably, PF+RWTV-DF resulted in boundarytracking errors on the order of 1-2 μm, which is comparable to the errorin mechanical pipette actuators.

The implemented edge reweighing strategy is adaptive by design, makingit very attractive in practice because it requires minimal parameterfine-tuning. The algorithm's performance was responsive to thesmoothness parameter (effective range being γ∈[2.0×10⁻⁴, 4.0×10⁻⁴]) andthe dynamics parameter (effective range being κ∈[10⁰, 10²]), yet notover-sensitive: the inventive algorithm's superior results were fromfixed parameters (γ, κ) across all trials, while other algorithmsrequired exhaustive two-parameter sweeps (β,γ) for each synthetic videotrial.

FIG. 12 shows one representative trial (Trial #4) in the time-series toobserve the qualitative differences between the algorithms. It wasobserved that the PF+RWTV-DF reconstruction has a fairly flat magnitude(for pixel values within the cell) compared to the otherreconstructions. The proposed algorithm's inherent segmentationcapability under such high-interference synthetic data is apparent fromthese results. Although PF+RWTV-DF reconstruction lost some cell details(i.e., in an ι2 reconstruction sense), the reduction in surroundinginterference served to improve the boundary identification at thesegmentation stage. On the other hand, the other algorithms producedreconstructions that contained significant surrounding interference,resulting in distorted edges in their subsequent segmentations (that mayrequire further image processing to remove).

Segmentation of Real Data

In addition to the synthetic data where ground truth was known, thealgorithms were also tested on in vitro DIC microscopy imagery of rodentbrain slices from a setup. For each algorithm, a parameter search wasperformed using a brute-force search to find the parameters that bestsegmented the cell by means of visual judgement. The DIC EPSF wasestimated visually to be σ_(d)=0.5, θ_(d)=2250°.

In FIGS. 13A-13C, three particularly challenging cell samples wereselected with respect to the amount of observable interference aroundand within the cell. A fixed global threshold was applied to each patchfor segmentation. The first example (FIG. 13A, from the top)demonstrated that reconstructions of other algorithms (compared to theproposed algorithm) were characteristically not piecewise smooth.Therefore, even in a case of moderate difficulty such as this one, thesealgorithms produce images with rounded edges which are not ideal forsegmentation. The second example (FIG. 13B) illustrated difficulty insegmenting cells with significant interference along the cell's edges,around the outside of the cell, and within the body of the cell. In thiscase, only the proposed algorithm is able to produce a cleansegmentation, especially along the cell's edges. Moreover, otheralgorithms reproduce the heavy interference scattered around the outsideof the cell and this requires subsequent image processing to remove. Thethird example (FIG. 13C) was a very difficult case where interferenceoccurs not only as distorted boundaries, but also as a close neighboringcell. All other algorithms visibly performed poorly while the proposedalgorithm remained fairly consistent in its performance.

In FIG. 14, snapshots from a video of a cell undergoing patch clampingwas compared with and without the proposed pipette removal method. Thepipette (and its respective mask) was tracked using a template matchingalgorithm via a user-selected template of the pipette. In FIG. 14A, thepipette did not cause interference when it was relatively far away fromthe cell. As the pipette approached the cell in FIG. 14B, interferencebegan to enter the deconvolution when inpainting was not applied.Without inpainting, an obfuscation between the pipette and the cell wasobserved in the deconvolution even when the pipette was not overlappingwith the cell. In FIG. 14C, the pipette was seen to be overlapping thecell image. Without inpainting, this overlap caused such severeinterference that attempting a pipette removal at the segmentation stageis clearly non-trivial. With inpainting, the deconvolution was shown toeffectively suppress the interfering pipette in all three snapshots.

Conclusion

Herein is described a new deconvolution algorithm to locate cellboundaries with high precision in DIC microscopy images of brain slices.In summary, some of the technical contributions of this algorithm are:(1) a PE step that is a computationally cheap and effective way ofremoving heavy organic interference with spectral characteristics, (2) adynamical ι1 reweighing approach for the propagation of second-orderedge statistics in online DIC cell segmentation, and (3) an inpaintingapproach for pipette removal that is possible with little modificationdue to the inherently flexible framework of the algorithm. Toquantitatively validate the performance of segmentation algorithms, thisExample also describes the novel adaptation of cell simulationtechniques to the specific data statistics of DIC microscopy imagery ofbrain slices in a publicly available MATLAB toolbox.

The proposed algorithm achieves state-of-the-art performance in trackingthe boundary locations of cells, with the average and MBEs achieving thedesired tolerances (i.e., 1-2 m) driven by the accuracy of actuatorsused for automatic pipette movement. These results led to the conclusionthat accurate visual guidance can be possible for automated patch clampsystems, resulting in a significant step toward high-throughput brainslice electrophysiology. A possible shortfall of the proposed algorithmarises from the implicit assumption that dynamics are spatially limited.In order for dynamics to positively contribute to the deconvolutionprocess, edges in the current frame should fall within the vicinity ofthe previous frame's edges; specifically, it should be bounded by thesupport of the prediction kernel as described by Equation 10.Improvements to the inventive method could include developing areal-time numerical implementation of this algorithm usingparallelization methods such as alternating direction methods ofmultipliers (ADMM), exploring the inclusion of shape-regularizers intothe optimization, investigating alternative methods that incorporatedynamics in a fashion that is not spatially-limiting, and characterizingdynamical functions that take into account physical models of motion inthe system (e.g., motion induced by the pipette, fluid dynamics, celldeformation physics).

Simulation Framework

Simulation Stage 1: Cell Shape Generation

The cell-shape is unique to different applications and can play asignificant role in algorithm development, necessitating customizationin cell simulations. Extensive work on generic shape representationdemonstrated that applying principal components analysis (PCA) oncell-shape outlines can be an excellent strategy for reconstructing cellshapes. Herein, an approach to generate synthetic cell-shapes using PCAand multivariate kernel density estimation sampling on subsampled cellcontours was applied. For shape examples, hand drawn masks of neuronsfrom DIC microscopy images of rodent brain slices were used.

Example cell-shapes were collected such that K coordinates were obtainedin clockwise continuous fashion (around the contour), beginning at thenorth-most point. These points were centered such that the centroid isat the origin. The (x, y) coordinates were concatenated into an

^(d) vectorx _(i)=(x ₁ , . . . ,x _(K) ,y ₁ , . . . y _(K))^(T)  (13)

where d=2K. This vector was then normalized via {circumflex over(x)}_(i)=x_(i)/∥x_(i)∥₂. All N normalized examples were gathered intothe following matrix X=({circumflex over (x)}₁, . . . , {circumflex over(x)}_(N)).

Eigen-decomposition was performed on the data covariance matrix formedas S=XX^(T)=V∧Y^(T). Cell-shapes may thus be expressed with thecoefficient vector b_(i) and the relationship given byx _(i) Vb _(i) ⇔b _(i) =V ^(T) x _(i)  (14)

Since PCA guarantees that cov(b_(i), b_(j))=0, ∀_(i)≠j, kernel densityestimation was performed individually on each of the coefficients toestimate its underlying distributions. This allows users to randomlysample from these distributions to produce a synthetically generatedcoefficient vector, {tilde over (b)}. The cell-shape may then betrivially converted into coordinates using the relationship given inEquation 14. A rotation (θ_(rot)) and scaling (γ_(scale)) are added tothe cell-shapes where necessary. FIGS. 15A-15B show a sampling ofsynthetic and actual cell-shapes extracted from the data.

1) Textured OPL Pixel-Space Embedding: The previously generatedcell-shape was embedded into the pixel space f_(x,y) by a texturegeneration method similar to methods found in other fluorescencemicroscopy simulators. In this stage, the cell-shape was embedded intothe pixel space with the generation of a textured OPL image. Perlinnoise is a well-established method for generating synthetic celltextures in fluorescent microscopy, and a similar concept was appliedbecause it generated realistic looking cell textures.

First, a binary mask, m(x, y), was generated using the cell-shape'scoordinates as polygon vertices and cast into an M×M image. Next, atextured image, t(x, y) was generated by frequency synthesis; a 2Dfilter with a 1/f^(b) frequency spectra was applied to white Gaussiannoise, where p is a persistence term which controls the texture'sheterogeneity. t(x, y) is rescaled such that {circumflex over(t)}(x,y)∈[0,1]. Finally, the OPL image is constructed byf(x,y)=(m(x,y)*h(x y))·(m(x,y)·{circumflex over (t)}(x,y))  (15)

where * signifies 2D convolution, and h(x, y) is a 2D filter (e.g., acircular pillbox averaging filter) that rounds the edges of the OPLimage. An example surface texture is simulated and shown in FIG. 7B.

Simulation Stage 2: Optical Imaging

Microscopy optics was modeled here using two linear components: aneffective point spread function (EPSF) and a lighting bias. DICmicroscopy exploits the phase differential of the specimen to derive theedge representations of microscopy objects. The EPSF is approximated asa convolution of the OPL image f (x, y) against the steerablefirst-derivative of Gaussian kernel, in Equation 1.

Nonuniform lighting of a microscope often causes a pronounced lightingbias in the image. A linear approximation of quadratic coefficientssufficiently expresses such a bias:b(x,y)=p ₀ +p ₁ x+p ₂ y+p ₃ x ² +p ₄ xy+p ₅ y ²  (16)

An example noiseless simulated image with the optical imaging model(including EPSF convolution and lighting bias) is shown in FIG. 7C.Polynomials p₀, . . . , p₅ are estimated from a randomly extracted patchfrom an actual DIC image using a least-squares framework.

Simulation Stage 3: Noise Generation

1) Synthetic Noise Generation Framework: Noise in a video frame may belinearly decomposed into organic and sensor components:n _(k)(x,y)=n ^(organic)(x,y)+n _(k) ^(sensor)(x,y)  (17)

with k denoting the frame index in time. Define n^(organic)(x, y) as thecomponents comprising of organic contributions in the specimen (e.g.,cellular matter, fluids) that remain static frame-to-frame, while n_(k)^(sensor)(x, y) (refers to noise from the CCD that is iid across everypixel and frame. For the purpose of statistical estimation, sequences ofimage frames {y_(k)(x, y)}_(k=1, . . . , K) that represented noise only(i.e., no target cell) with no tissue motion from pipette insertion wereselected. The organic noise component was estimated by averaging aseries of image frames (to reduce sensor noise):

$\begin{matrix}{{{n^{organic}( {x,y} )} \approx {\overset{\_}{y}( {x,y} )}} = {\frac{1}{K}{\sum\limits_{k = 1}^{K}{{y_{k}( {x,y} )}.}}}} & (18)\end{matrix}$

Similarly, a sensor noise sample may be estimated by subtracting theframe-average from the individual frame:n _(k) ^(sensor)(x,y)≈y _(k)(x,y)− y (x,y)  (19)

2) Organic Noise: A spectral analysis framework was employed to modeland generate realistic looking interference caused by organic tissue.The RAPSD, P(f), was defined as an averaging of the power spectraldensity (PSD) magnitudes along a concentric ring (whose radius isproportional to frequency, denoted by (f) on a DC-centered spatial PSDFourier plot. The RAPSD of random noise images reveal spectralcharacteristics shown in FIG. 16D. Phase information is simulated byrandomly sampling from a uniform distribution Φ(m, n)˜Uniform([0, 2π]),with m, n representing the spatial coordinates in the DC-centeredFourier domain, while the magnitude information is composed as a meanspectra P(f) from the RAPSD of example image patches{P_(k)}_(k=1, . . . , K) as

$\begin{matrix}{{\overset{\_}{P}(f)} = {\frac{\sqrt{2}}{K}{\sum\limits_{k = 1}^{K}{{P_{k}(f)}.}}}} & (20)\end{matrix}$

The organic noise spectrum is thus described by its magnitude and phasecomponents as{tilde over (S)}(m,n)=∥ P √{square root over (m ² +n²)})∥exp(j·Φ(m,n)).  (21)

An inverse spatial Fourier transform on the organic noise spectrumyields a synthetic organic noise image:ñ ^(organic)(x,y)=

⁻¹ {{tilde over (S)}(m,n)}.  (22)

An example organic noise image patch ñ^(organic)(x, y). (shown in FIG.16A) exhibits visual similarities with a patch of organic noise from anactual DIC image (shown in FIG. 16B). Additionally, similarities wereobserved in pixel distributions between synthetic organic noise andactual organic noise, qualitatively as shown in FIG. 16C, as well asquantitatively via a Kolmogorov-Smirnov two-sample goodness-of-fit testat the 5% significance level (after normalization by their respectivesample standard-deviations).

3) Sensor Noise: The CCD sensor contributes a mixture of Poisson noiseand zero-mean white-Gaussian noise. For modeling simplicity, it wasassumed that the sensor's individual pixels are uncorrelated in time andspace, and generated usingn _(k) ^(sensor)(x,y)=A _(g) ·n _(g) +A _(p) ·n _(p)  (23)

where n_(g)˜Normal(0, σ_(g) ²) and n_(p)˜Poisson (Δ_(p)) are randomlygenerated intensity values, {Ag, Ap} are amplitude parameters, and{σ_(g), λ_(p)} are the Gaussian's standard deviation and the Poisson'smean parameters respectively.

Image and Video Synthesis

1) Image Synthesis: Each simulation frame is generated as a linearcombination of the various synthesized components

$\begin{matrix}{{y( {x,y} )} = {{\lbrack {10^{\chi/10}\frac{{{n( {x,y} )}}_{F}}{{{g( {x,y} )}}_{F}}} \rbrack \cdot {g( {x,y} )}} + {n( {x,y} )} + {b( {x,y} )}}} & (24)\end{matrix}$

where χ is a user-defined SNR (in dB), and where ∥·∥F is the Frobeniusnorm. Table I summarizes the user-defined parameters of this simulator.

2) Video Synthesis: During the patch clamp process, pipette motioncauses the cells to undergo overall translation (e.g., moving from leftto right with respect to the frame), and some sequence of dilation andcontraction. Videos of K image frames are generated to simulate motionof the cell (rather than the pipette itself) by evolving a singletextured OPL pixel-space embedding image over time using a series ofgeometric transformations. Specifically, we simulate adilation/contraction using MATLAB's barrel transformation function andapply geometric translation along a random linear path through thecenter of the image with a parabolic velocity profile (e.g., anacceleration followed by a deceleration). These transformations produceda set of frames {f_(k)(x, y)}_(k=1,2), . . . which are convolved in 2Dusing Equation 1 and synthesized using Equation 24 to generate a set oftime-varying observations {y_(k)(x, y)}_(k=1,2), . . . comparable to avideo sequence of DIC imagery from a patch clamp experiment. Examplesnapshots from a synthetic video is shown in FIG. 17.

Example Environment and Computer Architecture

FIG. 20 illustrates an example environment 2000 in which certain aspectsof the present disclosure may be implemented. The environment 2000includes a process system 2005 and a tissue sample 2095. One of ordinaryskill will recognize that this is merely an example environment, andaspects of the present disclosure may be applied outside of thepresently discussed context. The process system 2005 may include aprocess controller 2010, a DIC microscope 2020, a cell identificationand tracking processor 2030 (ID processor 2030), and an actuator 2040.An example computer architecture that may implement one or more aspectsof the process controller 2010, DIC microscope 2020, cell identificationand tracking processor 2030, and actuator 2040 are described below withreference to FIG. 21.

Process system 2005 is configured to perform a process on tissue sample2095. For example, process system 2005 may be configured to performautomatic cell patch clamp electrophysiology on the tissue sample 2095(e.g., brain tissue). However, one of ordinary skill will recognize thatthis is merely an example.

Process controller 2010 controls the behavior of the remainingcomponents of process system 2005. DIC microscope 2020 may captureimages of the tissue sample 2095 using DIC microscopy. One of ordinaryskill will recognize that this is merely an example, and additionaland/or alternative imaging techniques may be utilized. The DICmicroscope 2020 may provide images of the tissue sample to the processcontroller 2010 (e.g., in real-time). The process controller 2010provides the images to ID processor 2030.

ID processor 2030 may be configured to identify a boundary of a cell ofthe tissue sample within the images captured by DIC microscope 2020. Asnon-limiting examples, ID processor 2030 may perform patch tracking,deconvolution, and segmentation on the images to identify the cellboundary, for instance, as described above in greater detail. IDprocessor 2030 may provide the boundary information to the processcontroller 2010.

Process controller 2010 may then control actuator 2040 to move aninstrument to coordinates corresponding to the cell boundary. In somecases, process controller 2010 may convert the image locationinformation from ID processor 2030 to coordinates corresponding to thetissue sample.

Although process controller 2010, DIC microscope 2020, cellidentification and tracking processor 2030, and actuator 2040 aredepicted as separate, one of ordinary skill will recognize that variousfeatures and configurations of the process controller 2010, DICmicroscope 2020, cell identification and tracking processor 2030, andactuator 2040 may be combined into one or more physical or logicaldevices.

FIG. 21 illustrates an example computing device architecture than canimplement one or more aspects of the process system 2005, processcontroller 2010, DIC microscope 2020, cell identification and trackingprocessor 2030, and actuator 2040, and a method according to the presentdisclosure. In some embodiments, process controller 2010, DIC microscope2020, cell identification and tracking processor 2030, and actuator 2040may have fewer, alternative, or additional components as thatillustrated in FIG. 21.

The computing device architecture 2100 of FIG. 21 includes a centralprocessing unit (CPU) 2102, where computer instructions are processed,and a display interface 2104 that acts as a communication interface andprovides functions for rendering video, graphics, images, and texts onthe display. In certain example implementations of the disclosedtechnology, the display interface 2104 may be directly connected to alocal display, such as a touch-screen display associated with a mobilecomputing device. In another example implementation, the displayinterface 2104 may be configured for providing data, images, and otherinformation for an external/remote display 2150 that is not necessarilyphysically connected to the mobile computing device. For example, adesktop monitor may be used for mirroring graphics and other informationthat is presented on a mobile computing device. In certain exampleimplementations, the display interface 2104 may wirelessly communicate,for example, via a Wi-Fi channel or other available network connectioninterface 2112 to the external/remote display 2150.

In an example implementation, the network connection interface 2112 maybe configured as a communication interface and may provide functions fordigital virtual assistant using voice, rendering video, graphics,images, text, other information, or any combination thereof on thedisplay. In one example, a communication interface may include amicrophone, camera, serial port, a parallel port, a general-purposeinput and output (GPIO) port, a game port, a universal serial bus (USB),a micro-USB port, a high definition multimedia (HDMI) port, a videoport, an audio port, a Bluetooth port, a near-field communication (NFC)port, another like communication interface, or any combination thereof.In one example, the display interface 2104 may be operatively coupled toa local display, such as a touch-screen display associated with a mobiledevice or voice enabled device. In another example, the displayinterface 2104 may be configured to provide video, graphics, images,text, other information, or any combination thereof for anexternal/remote display 2150 that is not necessarily connected to themobile computing device. In one example, a desktop monitor may be usedfor mirroring or extending graphical information that may be presentedon a mobile device. In another example, the display interface 2104 maywirelessly communicate, for example, via the network connectioninterface 2112 such as a Wi-Fi transceiver to the external/remotedisplay 2150.

The computing device architecture 2100 may include a keyboard interface2106 that provides a communication interface to a keyboard. In oneexample implementation, the computing device architecture 2100 mayinclude a presence sensitive input interface 2108 for connecting to apresence sensitive display 2107. According to certain exampleimplementations of the disclosed technology, the presence sensitiveinput interface 2108 may provide a communication interface to variousdevices such as a pointing device, a touch screen, a depth camera,microphone, etc. which may or may not be associated with a display.

The computing device architecture 2100 may be configured to use an inputdevice via one or more of input/output interfaces (for example, thekeyboard interface 2106, the display interface 2104, the presencesensitive input interface 2108, network connection interface 2112,camera interface 2114, sound interface 2116, etc.) to allow a user tocapture information into the computing device architecture 2100. Theinput device may include a mouse, a trackball, a directional pad, atrack pad, a touch-verified track pad, a presence-sensitive track pad, apresence-sensitive display, a scroll wheel, a digital camera, a digitalvideo camera, a web camera, a microphone, a sensor, a smartcard, and thelike. Additionally, the input device may be integrated with thecomputing device architecture 2100 or may be a separate device. Forexample, the input device may be an accelerometer, a magnetometer, adigital camera, a microphone, and an optical sensor.

Example implementations of the computing device architecture 2100 mayinclude an antenna interface 2110 that provides a communicationinterface to an antenna; a network connection interface 2112 thatprovides a communication interface to a network. As mentioned above, thedisplay interface 2104 may be in communication with the networkconnection interface 2112, for example, to provide information fordisplay on a remote display that is not directly connected or attachedto the system. In certain implementations, camera interface 2114 acts asa communication interface and provides functions for capturing digitalimages from a camera. In certain implementations, a sound interface 2116is provided as a communication interface for converting sound intoelectrical signals using a microphone and for converting electricalsignals into sound using a speaker. In certain implementations, a soundinterface 2116 is utilized to capture voice inputs for consumption by ofother components connected to the BUS 2134. According to exampleimplementations, a random-access memory (RAM) 2118 is provided, wherecomputer instructions and data may be stored in a volatile memory devicefor processing by the CPU 2102.

According to an example implementation, the computing devicearchitecture 2100 includes a read-only memory (ROM) 2120 where invariantlow-level system code or data for basic system functions such as basicinput and output (I/O), startup, or reception of keystrokes from akeyboard are stored in a non-volatile memory device. According to anexample implementation, the computing device architecture 2100 includesa storage medium 2122 or other suitable type of memory (e.g. such asRAM, ROM, programmable read-only memory (PROM), erasable programmableread-only memory (EPROM), electrically erasable programmable read-onlymemory (EEPROM), magnetic disks, optical disks, floppy disks, harddisks, removable cartridges, flash drives), where the files include anoperating system 2124, application programs 2126 (including, forexample, a web browser application, a widget or gadget engine, and orother applications, as necessary) and data files 2128 are stored.According to an example implementation, the computing devicearchitecture 2100 includes a power source 2130 that provides anappropriate alternating current (AC) or direct current (DC) to powercomponents.

According to an example implementation, the computing devicearchitecture 2100 includes a telephony subsystem 2132 that allows thecomputing device to transmit and receive sound over a telephone network.The constituent devices and the CPU 2102 communicate with each otherover a bus 2134.

According to an example implementation, the CPU 2102 has appropriatestructure to be a computer processor. In one arrangement, the CPU 2102may include more than one processing unit. The RAM 2118 interfaces withthe computer BUS 2134 to provide quick RAM storage to the CPU 2102during the execution of software programs such as the operating systemapplication programs, and device drivers. More specifically, the CPU2102 loads computer-executable process steps from the storage medium2122 or other media into a field of the RAM 2118 to execute softwareprograms. Data may be stored in the RAM 2118, where the data may beaccessed by the computer CPU 2102 during execution.

The storage medium 2122 itself may include a number of physical driveunits, such as a redundant array of independent disks (RAID), a floppydisk drive, a flash memory, a USB flash drive, an external hard diskdrive, thumb drive, pen drive, key drive, a high-density digitalversatile disc (HD-DVD) optical disc drive, an internal hard disk drive,a Blu-Ray optical disc drive, or a holographic digital data storage(HDDS) optical disc drive, an external mini-dual in-line memory module(DIMM) synchronous dynamic random access memory (SDRAM), or an externalmicro-DIMM SDRAM. Such computer readable storage media allow a computingdevice to access computer-executable process steps, application programsand the like, stored on removable and non-removable memory media, tooff-load data from the device or to upload data onto the device. Acomputer program product, such as one utilizing a communication systemmay be tangibly embodied in storage medium 2122, which may include amachine-readable storage medium.

According to one example implementation, the term computing device, asused herein, may be a CPU, or conceptualized as a CPU (for example, theCPU 2102 of FIG. 21). In this example implementation, the CPU may becoupled, connected, and/or in communication with one or more peripheraldevices, such as display. In another example implementation, the termcomputing device, as used herein, may refer to a mobile computing devicesuch as a smart phone, tablet computer, or smart watch. In this exampleimplementation, the computing device may output content to its localdisplay and/or speaker(s). In another example implementation, thecomputing device may output content to an external display device (e.g.,over Wi-Fi) such as a TV or an external computing system.

In example implementations of the disclosed technology, a computingdevice may include many hardware and/or software applications that areexecuted to facilitate the operations. In example implementations, oneor more I/O interfaces may facilitate communication between thecomputing device and one or more input/output devices. For example, auniversal serial bus port, a serial port, a disk drive, a CD-ROM drive,and/or one or more user interface devices, such as a display, keyboard,keypad, mouse, control panel, touch screen display, microphone, etc.,may facilitate user interaction with the computing device. The one ormore I/O interfaces may be used to receive or collect data and/or userinstructions from a wide variety of input devices. Received data may beprocessed by one or more computer processors as desired in variousimplementations of the disclosed technology and/or stored in one or morememory devices.

One or more network interfaces may facilitate connection of thecomputing device inputs and outputs to one or more suitable networksand/or connections; for example, the connections that facilitatecommunication with a number of sensors associated with the system. Theone or more network interfaces may further facilitate connection to oneor more suitable networks; for example, a local area network, a widearea network, the Internet, a cellular network, a radio frequencynetwork, a Bluetooth enabled network, a Wi-Fi enabled network, asatellite-based network, a wired network, a wireless network, etc., forcommunication with external devices and/or systems.

While several possible embodiments are disclosed above, embodiments ofthe present disclosure are not so limited. These exemplary embodimentsare not intended to be exhaustive or to unnecessarily limit the scope ofthe disclosure, but instead were chosen and described in order toexplain the principles of the present disclosure so that others skilledin the art may practice the disclosure. Indeed, various modifications ofthe disclosure in addition to those described herein will becomeapparent to those skilled in the art from the foregoing description.

Such modifications are intended to fall within the scope of the appendedclaims.

Any patents, applications, publications, test methods, literature, andother materials cited herein are hereby incorporated by reference intheir entirety as if physically present in this specification.

What is claimed is:
 1. A system of determining a cell edge for detectingone or more characteristics of the cell comprising: memory configured tostore data representative of a temporal sequence of images, the temporalsequence of images including at least: a current image containing atleast a portion of a cell; and a previous image containing at least aportion of the cell imaged before the current image; and one or moreprocessors coupled to the memory configured to: iteratively deconvolvethe current image of the temporal sequence of images; whereiniteratively deconvolving the current image identifies an edge of thecell within the current image based on a determined edge of the cellidentified in the previous image.
 2. The system of claim 1 furthercomprising a detecting module that receives data representative of theidentified edge of the cell within the current image and detects thecharacteristic of the cell.
 3. The system of claim 2, wherein thecharacteristic of the cell is selected from the group consisting of anelectrical activity, a molecular activity, a drug screening property,cell type, a biophysical property, a morphological property and agenetic property.
 4. The system of claim 1 further comprising: animaging device; and a detecting module; wherein the one or moreprocessors are further configured to: receive the data representative ofthe temporal sequence of images; and segment the deconvolved currentimage to determine the edge of the cell within the current image; andwherein the detecting module receives data representative of thedetermined edge of the cell within the current image and detects thecharacteristic of the cell.
 5. The system of claim 4, wherein thereceived image data representative of the current image of the temporalsequence of images further contains noise; wherein the one or moreprocessors are further configured to filter at least a portion of thenoise from the current image; and wherein iteratively deconvolvingcomprises iteratively deconvolving the filtered current image.
 6. Thesystem of claim 5, wherein the one or more processors are furtherconfigured to identify, within the filtered current image, a subset ofthe filtered current image containing at least a portion of the cell;and wherein iteratively deconvolving is only performed on the identifiedsubset of the filtered current image.
 7. The system of claim 5, whereinthe temporal sequence of images include the filtered current image, theprevious image, and a next image imaged after the filtered currentimage; and wherein the one or more processors are further configured to:set the filtered current image as the previous image; set the next imageas the current image; and repeat the filtering, iterativelydeconvolving, and segmenting.
 8. The system of claim 7, wherein: thetemporal sequence of images comprises a live video in real-time; and theedge of the cell within the filtered current image is determined in atleast near real-time.
 9. The system of claim 5, wherein the one or moreprocessors are further configured to iteratively deconvolve the filteredcurrent image by: determining an estimated edge that minimizes a costfunction; adjusting one or more weights of the cost function based onthe estimated edge; and repeating the determining and adjusting.
 10. Thesystem of claim 5, wherein the one or more processors are furtherconfigured to output positional data to an instrument proximal to thedetermined edge of the filtered current image.
 11. The system of claim 5further comprising an instrument; wherein the one or more processors arefurther configured to output positional data to the instrument.
 12. Thesystem of claim 5 further comprising an instrument selected from thegroup consisting of an electrode, an injector, and a manipulationinstrument; wherein the one or more processors are further configured tooutput positional data to the instrument proximal to the determined edgeof the filtered current image.
 13. The system of claim 9, wherein: thecost function comprises: a first term corresponding to a predictiveerror between the filtered current image and an image predicted from theestimated edge; and a second term corresponding to a connectivitydetermination of the estimated edge within the filtered current image;at least one adjusted weight of the cost function modifies theconnectivity determination of the estimated edge within the filteredcurrent image; and the determined edge of the cell within the previousimage impacts at least one adjusted weight.
 14. A computer-implementedmethod of determining a cell edge for detecting one or morecharacteristics of the cell comprising: iteratively deconvolving, by asystem comprising one or more processors, a current image of a temporalsequence of images, the temporal sequence of images including at least:the current image containing at least a portion of a cell; and aprevious image containing at least a portion of the cell imaged beforethe current image; and identifying, by the system, an edge of the cellwithin the current image based on a determined edge of the cellidentified in the previous image.
 15. The method of claim 14 furthercomprising: receiving, by the system, from an imaging device image datarepresentative of the temporal sequence of images prior to iterativelydeconvolving the current image, wherein the image data representative ofthe current image further contains noise; filtering, by the system, atleast a portion of the noise from the current image, wherein iterativelydeconvolving comprises iteratively deconvolving the filtered currentimage; and segmenting, by the system, the deconvolved filtered currentimage to determine the edge of the cell within the filtered currentimage.
 16. The method of claim 15 further comprising iterativelydeconvolving the filtered current image by: determining, by the system,an estimated edge that minimizes a cost function; adjusting, by thesystem, one or more weights of the cost function based on the estimatededge; and repeating, by the system, the determining and adjusting. 17.The method of claim 15 further comprising deconvolving the filteredcurrent image by: iteratively, by the system, estimating an edge thatminimizes a cost function utilizing alternating direction method ofmultipliers (ADMM); adjusting, by the system, one or more weights of thecost function based on the estimated edge; and repeating, by the system,the estimating and adjusting.
 18. The method of claim 15 furthercomprising moving an instrument proximal to the determined edge of thecell.
 19. A non-transitory computer-readable medium having storedthereon computer-readable instructions executable to cause a computer toperform a method of determining a cell edge for detecting one or morecharacteristics of the cell, the method comprising: receiving from animaging device image data representative of a temporal sequence ofimages including at least: a current image containing noise and at leasta portion of a cell; and a previous image containing at least a portionof the cell imaged before the current image; filtering at least aportion of the noise from the current image; iteratively deconvolvingthe filtered current image to identify an edge of the cell within thefiltered current image based on a determined edge of the cell within theprevious image; and segmenting the deconvolved filtered current image todetermine the edge of the cell within the filtered current image. 20.The computer-readable medium of claim 19, wherein the method furthercomprises deconvolving the filtered current image by: iterativelyestimating an edge that minimizes a cost function utilizing alternatingdirection method of multipliers (ADMM); adjusting one or more weights ofthe cost function based on the estimated edge; and repeating theestimating and adjusting.